Skip to content

Correct logic for WIEC app - #302

Open
mroda88 wants to merge 3 commits into
prep-release/fddaq-v5.7.0from
mroda/wiec_fix
Open

Correct logic for WIEC app#302
mroda88 wants to merge 3 commits into
prep-release/fddaq-v5.7.0from
mroda/wiec_fix

Conversation

@mroda88

@mroda88 mroda88 commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

All started at EHN1, when we realised we needed to readout a CRP with half of the FEMBs, because the CRP has only one CRU. We created a configuration that disabled the correct DetStreams. Speicifically, it's a configuration in which instead of 6 wibs, 2 wibs were completely disabled, 2 wibs were completely enabled and 2 wibs had 2 fembs enabled and 2 disabled.

When trying to run, we got an error from the HermesControllerModule, specifically this error here: https://github.com/DUNE-DAQ/hermesmodules/blob/dc08466de673700593d6d6431277de15aaec7a6c/plugins/HermesModule.cpp#L126  with the following  message:

Number of links in configuration (1) and firmware (2) don't matchand we got it for the two wibs which were only half enabled.

We spoke with alessandro and he suggested that the wrong part is here in appmodel, in the WIECApplication::generate_modules(). This fixes the problem.

Type of change

  • Documentation (non-breaking change that adds or improves the documentation)
  • New feature or enhancement (non-breaking change which adds functionality)
  • Optimization (non-breaking change that improves code/performance)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (whatever its nature)

While fixing, the code has been heavily commented to make sure we understand the subtleties of this particular case.

Testing checklist

  • Unit tests pass (e.g. dbt-build --unittest)
  • Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
  • Full set of integration tests pass (dunedaq_integtest_bundle.sh)
  • Python tests pass if applicable (e.g. python -m pytest)
  • Pre-commit hooks run successfully if applicable (e.g. pre-commit run --all-files)

This can only be tested with real hardware.

Further checks

  • Code is commented where needed, particularly in hard-to-understand areas
  • Code style is correct (dbt-build --lint, and/or see https://dune-daq-sw.readthedocs.io/en/latest/packages/styleguide/)
  • If applicable, new tests have been added or an issue has been opened to tackle that in the future.
    (Indicate issue here: # (issue))

@mroda88 mroda88 added the bug Something isn't working label Sep 3, 2026
@mroda88

mroda88 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Result of the cross talking test.
I created 2 configurations using crp4 at EHN1. In one configuration, called first_half I excluded the following:

<ref class="HermesDataSender" id="hds-np02-wib-1001-10g-0"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1001-10g-1"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1002-10g-0"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1002-10g-1"/>
  <ref class="DetectorStream" id="DetStream-420"/>
  <ref class="DetectorStream" id="DetStream-421"/>
  <ref class="DetectorStream" id="DetStream-422"/>
  <ref class="DetectorStream" id="DetStream-423"/>
  <ref class="DetectorStream" id="DetStream-428"/>
  <ref class="DetectorStream" id="DetStream-429"/>
  <ref class="DetectorStream" id="DetStream-430"/>
  <ref class="DetectorStream" id="DetStream-431"/>

while in a second configuration, called second_half I excluded the following:

<ref class="HermesDataSender" id="hds-np02-wib-1005-10g-1"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1006-10g-0"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1006-10g-1"/>
  <ref class="HermesDataSender" id="hds-np02-wib-1005-10g-0"/>
  <ref class="DetectorStream" id="DetStream-416"/>
  <ref class="DetectorStream" id="DetStream-417"/>
  <ref class="DetectorStream" id="DetStream-418"/>
  <ref class="DetectorStream" id="DetStream-419"/>
  <ref class="DetectorStream" id="DetStream-424"/>
  <ref class="DetectorStream" id="DetStream-425"/>
  <ref class="DetectorStream" id="DetStream-426"/>
  <ref class="DetectorStream" id="DetStream-427"/>

I started a run with the first half configuration. With the changes in this PR, all was ok. Here are the dashboards.

After a few minutes, from another terminal I started the second_half run. Boot was ok. Conf is the command that failed, as expected.

On the dashboard, from the first half session, the reported warning was from the readout application:

SourceID[subsystem: Detector_Readout id: 424] Request timed out for trig/seq_num 20.0, run_num 45192, window begin/end 111809771904871169/111809771905183669, data_destination: **internal_fragments_**

The frontend dashboard clearly showed a drop in packets and some of the queues reported 0 data.

On the dashboard, from the second half, the error was purely a drunc error.

root-controller is now in an error state

Looking in mode details, the application that caused the error was the readout application, not the WIEC application. The readout application complained because I was using the same receiver. The configuration, apart from the different excluded streams, was in fact identical.
Looking at the logs, it reported

2026-Sep-09 14:18:53,428 LOG [void dunedaq::dpdklibs::DPDKReaderModule::do_configure(const dunedaq::appfwk::DAQModule::CommandData_t&) at /tmp/root/spack-stage/spack-stage-dpdklibs-v2.3.10-25wt2vcwdq42iongtwik6iyxvbpo4mff/spack-src/plugins/DPDKReaderModule.cpp:113] datareader-runp02srv003eth0-0: Entering do_conf() method
2026-Sep-09 14:18:53,428 LOG [void dunedaq::dpdklibs::DPDKReaderModule::do_configure(const dunedaq::appfwk::DAQModule::CommandData_t&) at /tmp/root/spack-stage/spack-stage-dpdklibs-v2.3.10-25wt2vcwdq42iongtwik6iyxvbpo4mff/spack-src/plugins/DPDKReaderModule.cpp:119] Setting up EAL with params from config.
2026-Sep-09 14:18:53,429 LOG [void dunedaq::dpdklibs::ealutils::init_eal(int, const char**) at /tmp/root/spack-stage/spack-stage-dpdklibs-v2.3.10-25wt2vcwdq42iongtwik6iyxvbpo4mff/spack-src/src/EALSetup.cpp:298] EAL init arguments: eal_cmdline --proc-type=primary -a 0000:17:00.1 -l 0,2,4,6,58,60,62 --file-prefix=0000:17:00.1
EAL: Detected CPU lcores: 112
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Cannot create lock on '/run/user/103984/dpdk/0000:17:00.1/config'. Is another primary process running?
EAL: FATAL: Cannot init config
EAL: Cannot init config
EAL: Error - exiting with code: 1
  Cause: ERROR: EAL initialization failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@mroda88 mroda88

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mroda88 @dunedaqbot